{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 11 - Comparing distributions visually \n",
    "\n",
    "This lab will use two days worth of data from the 311 Service Request dataset on NYC Open Data. \n",
    "\n",
    "We will download the 311 data from March 3 and 4, 2019:\n",
    "1. Go to https://nycopendata.socrata.com/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9 and click \"View Data\". \n",
    "2. We will filter the data to only contain complaints made on March 3 or 4, 2019: \n",
    "    * If necessary, click on \"Filter\", and then click on \"Add new filter condition\". \n",
    "    * Select the column \"Created Date\" and change \"is\" to \"is between\". \n",
    "    * For the first date, select 03/03/2019 12:00:00 AM \n",
    "    * For the second date, select 03/05/2019 12:00:00 AM \n",
    "    * Check the box to left of the first date. The data should change, so that only those complaints created on March 3 or 4, 2019 show. \n",
    "3. To download the filtered data, click Export, then CSV. \n",
    "4. If necessary, rename the file so it will be named differently than the previous 311 data file.\n",
    "5. Upload your new data file to Jupyter Hub.\n",
    "\n",
    "As usual we import the matplotlib and pandas packages and set plots to appear in Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load your new 311 data file into the dataframe `calls`.  Use the `parse_dates` parameter to store the `Created Date` column as a `DatetimeIndex` type instead of string (see Lab 2 if you forget how to do this).  In other words, this parameter tells Pandas that the `Created Date` column in actually a date/time and not just some random text.  Parsing the date takes some extra time, so we have been skipping this step if we are not using the date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>calls = pd.read_csv(\"../Data/Mar3_4_2019_311_Service_Requests.csv\",parse_dates=[\"Created Date\"])</code>\n",
    "</details>\n",
    "\n",
    "Display `calls` to check that it was created correctly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing complaint times on Sunday March 3 vs. Monday March 4\n",
    "\n",
    "First we are going to compare the times complaints were made on Sunday March 3 with the times complaints were made on Monday March 4.  We will use histograms plotted on top of each other to visualize and compare the two distributions.  Before starting, how do you think timing of complaints will differ between Sunday and Monday?\n",
    "\n",
    "First we will create a filter to find all complaints created on March 3.  Type `mar3_filter = calls[\"Created Date\"].dt.day ==3` below and run the code.  The `.dt.day` gets just the day from the `DatetimeIndex` type and `==3` checks if it is equal to 3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use your March 3 `mar3_filter` to create a new dataframe called `mar3_calls` containing just the complaints that match this filter.  \n",
    "\n",
    "<details> <summary>Hint:</summary>\n",
    "    See Lab 9 or remember that we select from a dataframe using square brackets.  ex. `df[\"column_name\"]` selects the column `column_name` from the dataframe `df`.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar3_calls = calls[mar3_filter]</code>\n",
    "</details>\n",
    "\n",
    "Now create a dataframe called `mar4_calls` containing only the calls from March 4.  That is, create the filter and then use it to select from the `calls` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar4_filter = calls[\"Created Date\"].dt.day ==4\n",
    "mar4_calls = calls[mar4_filter]</code>\n",
    "</details>\n",
    "\n",
    "We will now plot a histogram of the hours of the March 3 calls.  We can get a Series of the hour each March 3 call was created with `mar3_calls[\"Created Date\"].dt.hour`.  Type and run this below to check this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So to make a histogram of the hours, we can just type `mar3_calls[\"Created Date\"].dt.hour.hist()` below and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the distribution?  You may want to increase the number of bins to 24.\n",
    "\n",
    "To plot two histograms on the same graph, just put the two pieces of code one after another in the same cell and run it.  Replot the histogram of the hours the March 3 complaints were created below and along with the histogram of the hours of the March 4 complaints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar3_calls[\"Created Date\"].dt.hour.hist()\n",
    "mar4_calls[\"Created Date\"].dt.hour.hist()</code>\n",
    "</details>\n",
    "\n",
    "The first histogram (March 3) is in blue and the second (March 4) in orange.  Can you see all of both histograms?  What do you notice?\n",
    " \n",
    "Let's make this plot look nicer.  To make a histogram transparent, add the parameter `alpha = 0.5`.  You can also change the color of a histogram to red by adding the parameter `color = \"red\"`.  A list of the possible colors is [here](https://matplotlib.org/examples/color/named_colors.html).  \n",
    "\n",
    "Can you add axis labels and a title?  \n",
    "\n",
    "Finally, we can add a legend with `plt.legend([\"Sunday March 3\",\"Monday March 4\"])` (This legend assumes the first histogram is for March 3 and the second is for March 4.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice?  What differences are there between the distributions of the times of the 311 complaints on Sunday March 3 and Monday March 4?\n",
    "\n",
    "It looks like there are more complaints on March 4 overall.  So instead of comparing the absolute number of complaints at each time, it might be better to compare the proportion of complaints at each time.  We do this by adding the parameter `density = True` to each histogram function.  Try this below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar3_calls[\"Created Date\"].dt.hour.hist(bins = 24,alpha = 0.5,density = True)\n",
    "mar4_calls[\"Created Date\"].dt.hour.hist(bins = 24,alpha = 0.5,density = True)\n",
    "plt.title(\"311 complaints by hour\")\n",
    "plt.xlabel(\"Hour\")\n",
    "plt.ylabel(\"Proportion of complaints\")\n",
    "plt.legend([\"Sunday March 3\",\"Monday March 4\"])</code>\n",
    "</details>\n",
    "\n",
    "## Comparing complaint distribution by borough on March 3 vs. March 4\n",
    "\n",
    "Now we will compare the distribution of complaints by borough on Sunday March 3 vs. Monday March 4.  Do you think the distribution will change?  If so, how?\n",
    "\n",
    "Recall that we created `mar3_calls` as a dataframe of just the March 3 calls and `mar4_calls` as a dataframe of just the March 4 calls.\n",
    "\n",
    "First, get the value counts of the `Borough` column in `mar3_calls` and in `mar4_calls` and store these in the variables `mar3_borough_counts` and `mar4_borough_counts` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar3_borough_counts = mar3_calls[\"Borough\"].value_counts()\n",
    "mar4_borough_counts = mar4_calls[\"Borough\"].value_counts()</code>\n",
    "</details>\n",
    "\n",
    "Just like with the histograms, we can plot two bar charts on top of each other by putting the functions in the same cells.  Try to make an overlapping plot the two bar charts of the borough counts for March 3 and for March 4 below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar3_borough_counts.plot(kind = \"bar\")\n",
    "mar4_borough_counts.plot(kind = \"bar\")</code>\n",
    "</details>\n",
    "\n",
    "We can't see the difference between the two plots, so add a parameter to change the color of the March 3 one to blue and the color of the March 4 one to red (or whatever two colors you like).  Also, to further distinguish between the two plots, we can add the parameter `width = 0.75` to the March 3 bar chart and the parameter `width = 0.5` to the March 4 bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>mar3_borough_counts.plot(kind = \"bar\",color = \"blue\",width = 0.75)\n",
    "mar4_borough_counts.plot(kind = \"bar\",color = \"red\",width = 0.5)</code>\n",
    "</details>\n",
    "\n",
    "What do you think the `width` parameter does?  What do you notice about the graph?  Which day has more 311 calls?\n",
    "\n",
    "Since there are more 311 calls on Monday than on Sunday, it is hard to tell if the distribution between boroughs is different.  We can normalize the counts so that they are proportions instead of counts by dividing by the total number of calls on each day.  For example, to normalize the March 3 counts, we write the following.  Try adding the code to normalize the March 4 counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "normalized_mar3_borough_counts = mar3_borough_counts/mar3_calls.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>normalized_mar3_borough_counts = mar3_borough_counts/mar3_calls.shape[0]\n",
    "normalized_mar4_borough_counts = mar4_borough_counts/mar4_calls.shape[0]</code>\n",
    "</details>\n",
    "\n",
    "Finally, let's plot the normalized counts or proportions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>normalized_mar3_borough_counts.plot(kind = \"bar\",color = \"blue\",width = 0.75)\n",
    "normalized_mar4_borough_counts.plot(kind = \"bar\",color = \"red\",width = 0.5)</code>\n",
    "</details>\n",
    "\n",
    "Can you add a title, axis labels, and a legend?\n",
    "\n",
    "We can make all sorts of interesting comparisons with filters.  Here is one comparing the type of calls made at night (6pm - 6am) with the type of calls made during the day.  Try running the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "before_6am_filter = calls[\"Created Date\"].dt.hour < 6\n",
    "after_6pm_filter = calls[\"Created Date\"].dt.hour > 18\n",
    "\n",
    "night_calls = calls[before_6am_filter | after_6pm_filter]\n",
    "day_calls = calls[~before_6am_filter & ~after_6pm_filter]\n",
    "\n",
    "night_counts = night_calls[\"Complaint Type\"].value_counts()\n",
    "night_counts[night_counts >50].plot(kind = \"bar\",color = \"blue\",width = 0.75)\n",
    "day_counts = day_calls[\"Complaint Type\"].value_counts()\n",
    "day_counts[day_counts > 100].plot(kind = \"bar\",color = \"red\",width = 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Challenges\n",
    "- Plot overlapping histograms of the hours that residential noise and commerical noise complaints are made.  Residential noise complaints are listed as `Noise - Residential` under `Complaint Type` and commerical noise complaints are listed as `Noise - Commercial` under `Complaint Type`.\n",
    "- Plot overlapping bar charts of the borough distribution of no heat/hot water complaints (`HEAT/HOT WATER`) and another complaint of your choice.  Is there a different in the distributions?\n",
    "- Your choice!  Pick one variable to compare in two different situations.  You can find a list of the different types of information you can get from teh `DatetimeIndex` [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DatetimeIndex.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}